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Instructional Effectiveness of the Collections Reading/Language Arts Program 



A Study of the Instructional Effectiveness 
of Collections, a Harcourt Reading/Language Arts Program 

This report describes one of a series of pilot studies that were conducted to evaluate the 
instructional effectiveness of Collections, a Harcourt Reading/Language Arts program. 

Background Information 

Harcourt School Publishers asked the Educational Research Institute of America (ERIA) to 
conduct a series of independent pilot studies to evaluate the effectiveness of Collections. 
Harcourt School Publishers sought out volunteer teachers to participate in the study; the 
Department of Test Services assisted in designing, securing, and scoring the data collection 
instruments; and ERIA conducted the study and analyzed the data that were collected. The study 
described in this report was conducted in the fall of the 1999-2000 school year. 

Research Questions 

The following research questions guided the design, procedures, and data analysis of the pilot 
study: 

• Is Collections instructionally effective? 

• Do selected themes significantly increase students’ understanding of key reading skills, 
concepts, and strategies as measured by the program’s Reading Inventory 

• As measured by the program’s Reading Skills Tests? 

• As measured by the TAAS Test? 

• As measured by a standardized achievement test ( Stanford 9) ? 

Design and Procedures of the Study 

Because this study was focused on the primary levels of the program, it was limited to four 
grades - Kindergarten, grade 1, grade 2, and grade 3. 

Teachers who volunteered to participate in the study had not used the program previously. The 
teachers were encouraged to select two themes to pilot over a nine- to twelve-week period. They 
also agreed to administer data collection instruments before beginning instruction and again after 
completing instruction. 

A total of 40 teachers volunteered to participate in the study: 12 at Kindergarten; 8 at grade 1; 10 
at grade 2; and 10 at grade 3. The participating teachers came from 5 different schools in 4 
different districts in Texas. 
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A quasi -experimental pretest-posttest design was used. Before instruction began, students were 
administered one or more pretests depending on their grade level. The classroom teachers 
administered all tests. Table 1 summarizes the data collection instruments that were used. 

TABLE 1 

Data Collection Instruments 



Grade 


Pretests 


Posttests 


K 


• Collections K/I Reading Inventory 


• Collections K/I Reading Inventory 


I 


• Collections K/I Reading Inventory 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Complete Battery), Level P I/Form S 


• Collections K/I Reading Inventory 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Complete Battery), Level PI/Form S 


2 


• Collections 2/3 Reading Inventory 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level P I/Form S 


• Collections 2/3 Reading Inventory 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level PI/Form S 


3 


• Collections Reading Skills Tests 
(matched to each theme taught) 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level P2/Form S 

• TAAS Practice Test (Reading 
Comprehension) 


• Collections Reading Skills Tests 
(matched to each theme taught) 

• Reading Subtest of the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level P2/Form S 

• TAAS Practice Test (Reading 
Comprehension) 



Following the pretests, the teachers taught the selected themes using the resources and 
procedures contained in Collections. Teachers also received training from experienced 
consultants in methods of implementing and using the program. Instructional time varied from 
nine to twelve weeks depending on the themes taught. Collections was the reading/language arts 
program used by all of the teachers. Instruction included a broad sample of reading skills from 
the strands of decoding, vocabulary, comprehension, literary appreciation, and study skills. 

Upon completion of the selected themes, students were administered the posttests. All data 
collection instruments were returned to the Educational Research Institute of America where 
they were processed. The Stanford 9 Achievement Tests were scored at the Harcourt Educational 
Measurement Scoring Center. The Collections Reading Inventories and the Collections Skills 
Tests were scored at ERIA, and all of the data were analyzed by ERIA. 
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Findings 

Descriptions of each of the assessments used in the study and the assessment results are reported 
below: 

Content of the Kindergarten and Grade 1 Reading Inventory 



Kindergarten and Grade 1 
Reading Inventory 




Subtest 


Kindergarten 


Grade 1 No. 




No. of Items 


of Items 


Phonemic Awareness 






Detecting Rhyming Sounds 


5 




Detecting Beginning Sounds 


5 


5 


Detecting Ending Sounds 


5 


5 


Graphophonemic Knowledge 






Letter Recognition 


5 




Letter-Sound Relationships: Beginning 


5 


5 


Sounds 






Letter-Sound Relationships: Ending 


5 


5 


Sounds 






Letter-Sound Relationships: Medial Vowel 




5 


Sounds 






Initial Blend Substitution 




5 


Final Consonant Substitution 




5 


Final Blend Substitution 




5 


Medial Vowel Substitution 




5 


Word Recognition 


5 


5 


Writing (Spelling) 




5 


Writing Letters 


5 




Writing Words 


5 


5 


Reading Comprehension 




5 


TOTAL 


45 


65 
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t^ndergar® Test ResiilMi’. ' 

Table 2 summarizes the Pretest and Posttest means and standard deviations for the Kindergarten 
Reading Inventory. 

TABLE 2 



Kindergarten Results 



Kindergarten 


Reading Inventory (45 test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


3 


44 


25.77 


9.09 


Posttest 


7 


43 


29.42 


9.82 



At kindergarten the mean Reading Inventory test scores improved from 58% correct to 66% 
correct after instruction. This increase was highly significant (r=5.161; p<.0001). 



CrTade One Test Results 



Table 3 summarizes the Pretest and Posttest means and standard deviations for the Grade 1 
Reading Inventory. 



TABLE 3 
Grade 1 Results 



Grade 1 


Reading Inventory (65 test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


15 


64 


44.93 


11.65 


Posttest 


20 


64 


56.11 


7.68 



At Grade 1 the mean Reading Inventory test scores improved from 68% correct to 86% correct 
after instruction. This increase was highly significant (r=13.817; p<.0001). 
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TABLE 4 
Grade 1 Results 



Grade 1 


Stanfo 


ird Achievem 


lent Tests: Grade 1 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest: Sounds and 
Letters 


13 


40 


29.9 


6.6 


Posttest: Sounds 
and Letters 


8 


40 


35.1 


5.0 


Pretest: Word 
Reading 


10 


40 


26.8 


8.4 


Posttest: Word 
Reading 


12 


40 


33.5 


6.2 


Pretest: Sentence 
Reading 


5 


30 


17.6 


8.2 


Posttest: Sentence 
Reading 


3 


30 


22.5 


6.3 


Pretest: Total 
Reading 


34 


110 


74.1 


19.0 


Posttest: Total 
Reading 


23 


110 


91.0 


15.7 



A paired t-test for the three subtests and the total test score on the Stanford Achievement Tests: 
Grade One resulted in a significant increase (Sounds and Letters: r=10.035; p<.0001); (Word 
Reading: t=10.32; p<.0001); (Sentence Reading: t=6.76; p<.0001); (Total Reading: r=13.048; 

p<.0001). 

The percentage increase in Stanford Achievement Test Scores: Grade One from pretest to posttest 
were: Letters and Sounds from 73% to 85% correct; Word Reading from 68% to 85% correct; 
Sentence Reading from 60% to 77% correct; and Total Reading from 67% to 83% correct. 
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Grade TwoiTest Results 



Grade 2 Reading Inventory 
Subtests 


Grade 2 
No. of Items 


Graphophonemic Knowledge 




Initial Consonant Substitution 


5 


Initial Blend Substitution 


5 


Final Consonant Substitution 


5 


Final Blend Substitution 


5 


Medial Vowel Substitution 








Word Recognition 


10 






Writing (Spelling) 


10 






Reading Comprehension 


15 






Total 


55 



Table 5 summarizes the means and standard deviations of the scores on the pre-test and the post- 
test for the Grade 2 Reading Inventory. 



TABLE 5 
Grade 2 Results 



Grade Two 


Reading Inventory (55 test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


19 


55 


43.03 


7.72 


Posttest 


26 


55 


47.13 


5.64 



At grade 2 the mean Reading Inventory test scores improved from 78% correct to 85% correct 
after instruction. This increase was highly significant (t=7.098; p < .0001). 
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TABLE 6 



Grade 2 


Stanfo 


>rd Achieverr 


lent Tests: Grade 2 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest: Word Study 
Skills 


6 


20 


15.2 


3.4 


Posttest: Word 
Study Skills 


9 


20 


17.0 


3.1 


Pretest: Word 
Reading 


2 


20 


15.2 


3.9 


Posttest: Word 
Reading 


3 


20 


16.6 


3.1 


Pretest: Reading 
Comprehension 


11 


30 


24.4 


4.8 


Posttest: Reading 
Comprehension 


12 


30 


26.3 


4.7 


Pretest: Total 
Reading 


23 


70 


55.1 


10.9 


Posttest: Total 
Reading 


28 


70 


60.0 


9.6 



A paired t-test for the three subtests and the total test score on the Stanford Achievement Tests: 
Grade Two resulted in a significant increase (Word Study Skills: t=7.533', p<.0001); (Word 
Reading: r=4.789; p<.0001); (Reading Comprehension: t=5.476', p<.0001); (Total Reading: 
?=7.936; p<.0001). 

The percentage increase in Stanford Achievement Test Scores: Grade Two from pretest to 
posttest were: Word Study Skills from 76% to 85% correct; Word Reading from 76% to 83% 
correct; Reading Comprehension from 81% to 88% correct; and Total Reading from 79% to 86% 
correct. 



9 

er|c 



7 



Instructional Effectiveness of the Collections Reading/Lxinguage Arts Program 



jGrade Thcee T^t.Residts I i j 



Grade 3 Skills Tests 
Subtests 


Theme 1 
Test 


Theme 2 
Test 


Decoding Phonics 






Prefixes and Suffixes 


6 




Vocabulary 






Selection Vocabulary 


20 


16 


Vocabulary in Context 




6 


Reading Comprehension 






Predict Outcomes 


4 


4 


Draw Conclusions 






Literary Concepts 






Story Elements (Character, Setting Plot) 




4 








Total 


30 


30 



The TAAS Practice Test constructed for use in this study was based on the same TEKS 
objectives/standards as the actual TAAS tests. That is, it assesses the same comprehension skills 
using both fiction and non-fiction passages that are used on the TAAS tests. It is closely 
patterned after the TAAS tests that have been released by the Texas Education Agency. 

The TAAS Practice Test included a total of seven reading selections with six test items for each 
selection (a total of 42 reading comprehension test items). 

Table 7 summarizes the means and standard deviations of the scores on the pre-test and the post- 
test for the Grade 3 Theme 1 Assessment. 



TABLE 7 

Grade 3 Results: Theme 1 



Grade Three 


Theme 1 Assessment (3( 


test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


2 


30 


23.52 


7.13 


Posttest 


5 


30 


26.64 


4.90 



At grade 3 the mean Theme One Assessment scores improved from 78% correct to 87% correct 
after instruction. This increase was highly significant (t=8.346; p < .0001). 
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Table 8 summarizes the means and standard deviations of the scores on the pre-test and the post- 
test for the Grade 3 Theme 2 Assessment. 



TABLE 8 

Grade 3 Results: Theme 2 



Grade Three 


Theme 2 Assessment (3( 


test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


2 


30 


24.04 


6.97 


Posttest 


5 


30 


27.06 


4.10 



At grade 3 the mean Theme Two Assessment scores improved from 80% correct to 90% correct 
after instruction. This increase was highly significant (t=8.876; p < .0001). 



Table 9 summarizes the means and standard deviations of the scores on the pre-test and the post- 
test for the Grade 3 Theme 2 Assessment. 



TABLE 9 

Grade Three Results: TAAS Practice Tests 



Grade Three 


TAAS Practice Test (42 test items) 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest 


5 


42 


23.12 


8.05 


Posttest 


7 


42 


26.34 


6.85 



At grade 3 the mean TAAS Practice Test scores improved from 57% correct to 63% correct after 
■ instruction. This increase was highly significant (t=6.767; p < .0001). 
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TABLE 10 



Grade 3 


Stanfc 


ird Achievem 


lent Tests: Grade 3 




Lowest Score 


Highest Score 


Mean Score 


Standard 

Deviation 


Pretest: Word Study 
Skills 


6 


20 


14.8 


3.8 


Posttest: Word 
Study Skills 


5 


20 


16.3 


3.6 


Pretest: Reading 
Vocabulary 


3 


20 


14.4 


3.8 


Posttest: Reading 
Vocabulary 


5 


20 


15.7 


3.3 


Pretest: Reading 
Comprehension 


5 


29 


20.4 


6.1 


Posttest: Reading 
Comprehension 


7 


30 


22.4 


6.1 


Pretest: Total 
Reading 


18 


68 


49.6 


12.0 


Posttest: Total 
Reading 


20 


70 


60.0 


11.3 



A paired t-test for the three subtests and the total test score on the Stanford Achievement Tests: 
Grade Three resulted in a significant increase (Word Study Skills: r=6.936; p<.0001); (Reading 
Vocabulary: r=6.204; p<.0001); (Reading Comprehension: r=5.161; p<.0001); (Total Reading: 
r=8.626; p<.0001). 

The percentage increases in Stanford Achievement Test Scores: Grade Three from pretest to 
posttest were: Word Study Skills from 74% to 83% correct; Reading Vocabulary from 72% to 
79% correct; Reading Comprehension from 53% to 75% correct; and Total Reading from 71% to 
86% correct. 
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The percent of gains on each of the tests is shown in Table 1 1 : 

TABLE 11 



Summary of Test Score Increases 





Percent Correct 
on Pretest 


Percent Correct 
on Posttest 


Percent 
Gain from 
Pretest to 
Posttest 


Grade/Test 




Kindergarten Reading Survey 


58% 


66% 


-1-8% 


Grade 1 

Reading Survey 


68% 


86% 


+18% 


Stanford Achievement Test 
Grade 1: Total Score 


67% 


83% 


+16% 


Grade 2 

Reading Survey 


78% 


85% 


+1% 


Stanford Achievement Test 
Grade 2: Total Score 


79% 


86% 


+1% 


Grade 3 

Theme One Test 


78% 


87% 


+9% 


Grade 3 

Theme Two Test 


80% 


90% 


+\6% 


Grade 3 
TAAS Test 


57% 


63% 


+6% 


Stanford Achievement Test 
Grade 3: Total Score 


71% 


86% 


+15% 


Averages 


70% 


80% 


+10% 



Summary of Results 

• The consistent increases at every grade level from Kindergarten through Grade 3 are 
significant. 

• Significant increases were found for every test and every subtest at every grade level. This 
provides substantial evidence of continued improvement in reading and reading sub-skills for 
the students in this study. 

• Gains of this magnitude for such a brief period of instruction are quite remarkable 
considering that the teachers volunteered to teach the units and received only minimal in- 
service training. 

• The increase on the TAAS Practice Test scores after only two Instructional Units is also 
significant. Since this assessment consists only of reading comprehension test items, it is 
clear that the instructional units produced significant gains in students’ ability to read and 
comprehend — as well as with beginning reading skills. Increases of 10% on comprehension 
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tests are seldom found in the research literature. Skills tests often increase more significantly 
with concentrated focus than do comprehension scores. 

• Some of the results may have shown even greater gains. However, there was a ceiling effect 
for some of the assessments. (Students scored perfect or almost perfect scores on the posttests 
thus limiting the gain scores.) 

• The increases were greater for grade 1 than for grades 2 and 3. This result suggests (as do 
many studies of this sort) that getting programs established in the earliest grades provides the 
greatest possibility of success. 

• It is also significant that no test scores stayed the same. They all increased. In a short-term 
study of this sort it is not uncommon to note some test scores that do not increase at all. 
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